This Page Is Inserted by IFW Operations 
and is not a part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of 
the original documents submitted by the applicant. 

Defects in the images may include (but are not limited to): 

• BLACK BORDERS 

• TEXT CUT OFF AT TOP, BOTTOM OR SIDES 

• FADED TEXT 

• ILLEGIBLE TEXT 

• SKEWED/SLANTED IMAGES 

• COLORED PHOTOS 

• BLACK OR VERY BLACK AND WHITE DARK PHOTOS 

• GRAY SCALE DOCUMENTS 



IMAGES ARE BEST AVAILABLE COPY. 



As rescanning documents will not correct images, 
please do not report the images to the 
Image Problem Mailbox. 



(12) INTERNATIONAL APPUCATION PUBUSHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) Worid Intellectual Property Oi^anization 
International Bureau 

(43) International Publication Date 
31 January 2002 (31.01.2002) 




PCT 



(10) International Publication Number 

WO 02/08899 A2 



(51) InteniationalPateatClassificatioa^: G06F9/50 

(21) International Application Number: PCT/USOl/20977 

(22) iatemaCioaal Filing Date: 28 June 2001 (28.06.2001) 

(25) Filing Language: English 

(26) Publication Language: English 



(30) Priority Data: 
09/608,521 



30 June 2000 (30.06^000) US 



(71) Applicant: CEREVA NETWORKS, INC. [USAJSJ; 3 
Network Drive, Mariborough, MA 01752-3083 (US). 

(72) inventors: BOPARDIKAR, Raju, C; 1 10 Carlisle Pines 
Drive. Carlisle, MA 01741 (US). BAST, Jacob, Y.; 4 Tally 
Ho Lane, Framingham, MA 01701 (US). CARDONE, 
Gary, A.; 131 Hidden Valley Road, Groton, MA 01450 
(US). KAUFMAN, David, E.; 7 Clarice Street, Sharon, 
MA 02067 (US). MACEACHERN, Stuart, P.; 225 
Charleston Meadows Drive, Westboro, MA 01581 (US). 
MCLEOD, Bruce, D.; 24 Magnolia Road, Sudbuiy, MA 



01776 (US). NOLAN, James, M; 394 Gorwin Drive, 
Holliston, MA 01746 (US). RADOUCH, Zdeoek; 25 
Breamorc Road, Newton, MA 02458 (US). STIFFLER, 
Jack, 286 Delano Road, Marion, MA 02738 (US). 
WENTWORTH, James, A.; 10 Lcblanc Road, Shrcws- 
buiy. MA 01545 (US). 

(74) Agent: KUDIRKA, Paul, E.; Kodirica St Jobse. LLP, Suite 
1510, One State Street, Boston. MA 02109 (US). 

(81) Designated States (nationaJ): AE. AG, AU AM, AT, AU, 
AZ. BA. BB, BG, BR, BY, BZ, CA. CH, CN, CO, CR. CU, 
CZ. DE, DK, DM, DZ, EE, ES, FI, GB. GD, GE, GH, GM. 
HR, HU. ID. XL, m, IS, JP, KB, KG, KP, KR, KZ, LC, LK, 
LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW. MX. 
MZ, NO. NZ. PL, PT, RO. RU, SD, SE, SG, SI, SK, SL, 
TJ. TM, TR, XT. TZ, UA, UG. UZ, VN. YU. ZA, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KB, LS. MW, MZ, SD, SL, SZ, TZ, UG, ZW). Eurasian 
patent (AM, AZ, BY, KG. KZ, MD, RU, TJ. TM), European 
patent (AT. BE, CH, CY, DE, DK. ES. FI, FR, GB, GR, IE, 
IT. LU, MC, NTL, PT, SB, TR), OAPI patent (BF, BJ. CF. 
CG. CI, CM. GA, GN, GW, ML, MR. NE, SN. TD. TG). 

[Continued on next page J 



(54) Title: METHOD AND APPARATUS FOR IMPLEMENTING HIGH-PERFORMANCE, SCALEABLE DATA PROCF^S- 
ING AND STORAGE SYSTEMS 



I |c<-gyr| [cuEwT| [ojewtI icuoffl |cub<t1 

t ^ 104 + ♦ > 



^cuewt| {clcktI 



0\ 
00 

oo 

o 




(57) Abstract: A data system architecture is described that allows multiple processing and storage resources to be connected to 
multiple clients so as: 1) to distribute the clients'woikJoad efficiently across the available resources; and 2) to enable scaleable 
expansion, both in terms of the number of clients and in the number of resources. The major features of the architechue are separate, 
modular, client and resource elements that can be added independently, a high-performance cross-bar data switch interconnecting 
these various elements, separate serial communication paths for controlling the cross-bar switch settings, separate communication 
paths for passing control information among the various elements and a resource utilization methodology that enables clients to 
distribute processing or stomge tasks across all available resources, thereby eliminating *'hot spots** resulting from uneven utilization 
of those resources. 
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METHOD AND APPARATUS FOR IMPLEMENTING HIGH-PERFORMANCE, 
SCALEABLE DATA PROCESSING AND STORAGE SYSTEMS 

Field of the Invention 
This Invention relates to data processing systems and, In particular, to data 
processing systenis involving the transfer, manipulation, storage and retrieval of large 
amounts of data. 

Background of the Invention 

In data processing applications involving the transfer, manipulation, storage 
and retrieval of large amounts of data, the most serious performance limitations 
include (1) difficulties In moving data between users who need access to the data and 
resources used to store or process the data and (2) difficulties in efficiently distributing 
the workload across the available resources. These difficulties are particulariy 
apparent, for example, in disk-based storage systems in which the greatest 
performance limitation is the amount of time needed to access information stored on 
the disks. As databases increase In size, requiring more and more disks to store that 
data, this problem grows correspondingly worse and, as the number of users desiring 
access to that data increase, the problem is compounded even further. Yet the trends 
toward both laiger databases and an increased user population are overwhelmingly 
apparent, typified by the rapid expan^ton of the Internet. 

Cunrent techniques used to overcome these difficulties include reducing access 
time by connecting users to multiple resources over various types of high-speed 
communication channels (e.g., SCSI buses, fiber channels and Infiniband busses) and 
using caching techniques in an attempt to reduce the necessity of accessing tiie 
resources. For example, in the case of storage systems, large random-access 
memories are often positioned locally to the users and are used as temporary, or 
cache, memories that store the most recently accessed data. These cache memories 
can be used to eliminate the need to access the storage resource itself when the 
cached data is subsequentiy requested and tiiey thereby reduce ttie communication 
congestion. 

Various distribution algorithms are also used to allocate tasks among those 
resources in attempts to overcome the workload distribution problem, in all cases, 
however, data Is statically assigned to specific subs ts of th available resources. 
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Thus, when a resource subset temporarily becomes overioaded by multiple clients 
simultaneously attempting to access a relatively small portion of the entire system, 
performance is substantially reduced. Moreover, as the number of clients and the 
workload increases, the perfomnance rapidly degrades even further since such 
systems have limited scalability. 

Summary of the Invention 

In accordance with one illustrative embodiment of the invention, users are 
connected to access Interfaces. In turn, the access interfaces are connected to a pool 
of resources by a switch fabric. The access interfaces communicate with each client 
with the client protocol and then Interfaces with the resources In the resource pool to 
select the subset of the resource pool to use for any given transaction and distribute 
the woridoad. The access interfaces make it appear to each client that the entire set 
of resources Is available to it without requiring the client to be aware that that the pool 
consists of multiple resources. 

In accordance with one embodiment, a disk-based storage system is 
implemented by client interfaces refen-ed to as host modules and processing and 
storage resources refen^ed to as metadata and disk interface modules, respectively. 

The invention eliminates the prior art problems by enabling both client 
interfaces and processing and storage resources to be added independently as 
needed, by providing much more versatile communication paths between clients and 
resources and by allowing the workload to be allocated dynamically, with data 
constantly being directed to those resources that are currentiy least active. 

Brief Description of the Drawings 
The above and furttier advantages of the invention may be better understood 
by referring to the following descripti'on In conjunction witii the accompanying drawings 
in which: 

Figure 1 is a block schematic diagram of a resource access system constructed 
in accordance with the principles of the present Invention. 

Figure 2 is a block schematic diagram of an illustrative storage system 
embodiment implemented wiOi ttie architecture of Figure 1. 

Figure 3 is a detailed block sch matic diagram of a host Interface module. 
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Figures 4A-4C, when placed together, form a flowchart Hlustratlng the steps in a 
process canied out by the host interface module in response to a request from a 
client. 

Figure 5 is a detailed blocl^ diagram of a disk interface module. 

Figure 6 is a flowchart illustrating the processing steps perfomied by software 
running in the disk interface module. 

Figure 7 is a detailed bloclc schematic diagram of a metadata module. 

Figures 8A and 8B, when placed together, fomi a flowchart illustrating 
processing steps performed by software running in the metadata module. 

Figure 9 Is a detailed block schematic diagram of a switch module. 

Detailed Description 

A block schematic diagram of a resource access system 100 in accordance 
with an embo'diment of the invention is shown in Figure 1. The system consists of 
three components. Access interfaces 106-1 12 provide clients, such as client 102 and 
104, with access to the system 100 and provide other access-related resources. A 
pool of resources 1 18-124 may comprise, for example, data processing or storage 
devices. A switch fabric 1 14 and 1 16 Interconnects the access interfaces 106-1 12 
and the resources 1 18-124. Since the requirements for communicating control 
information differ significantly from those for data communication, the switch fabric 
consists of a control switch fabric 1 14 and a data switch fabric 1 16 in order to provide 
different paths and protocols for control and data. For example, control transfer 
protocols generally divide the control infomiation into relatively small packets that are 
transfenred using packet-switching technology. In contrast, data transfer protocols 
generally consist of larger packets conveyed over a circuit-switched fabric. The 
separation of the switch fabric Into two sections 114 and 116 allows each type of 
communication path to be optimized for its specific function and enables service 
requests to be transferred to a resource, via the control switch fabric 114 without 
interfering with the data transfemng capacity of the data switch fabric 116. 

In accordance with the principles of the invention, the access interfaces 106- 
1 12 operate to virtualize the pool of resources 1 1 8-124, thereby making it appear to 
each client, such as clients 102 and 104, that the entire set of resources 1 1 8-124 is 
available to it without requiring the client to be aware of the fact that that the pool is in 
fact partitioned Into multiple resources 118-124. 
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This virtualization is accomplished by enabling the access interfaces 106-1 12 to 
serve as communication protocol temifnators and giving them the ability to select the 
subset of the resource pool 1 1 8-1 24 to use for any given transaction. An access 
interface, such as Interface 106, is thus able to communicate with a client, such as 
client 102, using the client's protocol for messages. The interface 106 parses a 
message received from the client into a portion representing data and a portion 
consisting of commands or requests for sen/ice. The interface 106 then interprets 
those requests and distributes the woridoad and the associated data across the pool 
of resources 118-124. 

The distribution of the woridoad may entail accessing a number of resources by 
the access Interface and brings with It several major advantages. For example, it 
allows the woridoad to be distributed across the available resources, preventing the 
"hotspots" typically encountered when multiple clients independently attempt to access 
multiple resources. Since clients generally do not have knowledge about other clients' 
activities, It Is very difficult, if not impossible, for the clients themselves to achieve any 
such level of load balancing on their own. In addition, it enables resources to be 
added non-dismptiveiy to the resource pool. Clients need not be aware that additional 
resources have been made available since the access interfaces themselves are 
responsible for allocating resources to requests for service. This, in turn, allows the 
system capacity to be scaled to meet demand as that demand increases over time, 
Similariy, the ability of the access interfaces to distribute workloads allows the extemal 
connectivity to be increased to accommodate additional clients, again without 
dismpting on-going activities with existing dients. 

The Inventive system can be used to constnjct resource allocation systems for 
any type of resources. The remainder of this disclosure describes an embodiment 
which implements a disk-based storage system, but this embodiment should not be 
considered as limiting. In this embodiment, the access interfaces 106-112 are 
refenred to as "host Interface modules" and the resources are disk storage devices. 
The disk storage devices are connected to the switch fabric by "disk interface 
modules" and separate processing modules called "metadata" modules are also 
provided. 

The storage system embodiment is shown in Figure 2. The storage system 
200 consists of a set of access modules 206-210 called host interface modules, and 
two types of resource modules: disk interfac modules 21 8*222 and metadata 
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modules 212-214. The host interface modules 206-210 provide one or more clients, 
of which clients 202 and 204 are shown, access to the system 200 and communicate 
with each client 202, 204 using the dienf s message passing protocol. The host 
interface modules 206-210 parse requests from the clients 202, 204 for disk and file 
system accesses and distribute the storage load across the entire set of disks 
connected to the system 200, of which disks 226-230 are shown. The host interface 
modules are responsible for the logical allocation of the storage resources. 

The disk interface modules 218-222 each support up to 450 disks and are 
responsible for the physical allocation of their disk resources. The disk interface 
modules provide data buffering, parity generation and checking and respond to 
requests from the host interface modules for access to their associated disks. 

The metadata modules 212-214 provide a processing resource that maintains 
the structure and consistency of file systems used in the system. They are used when 
the storage system serves as a standalone file system, for example, in a networi<ed 
environment, and hence assumes the responsibility for maintaining the file systems. 
The data used to describe the objects In the file system, their logical locations, 
relationships, properties and stmctures, is called "metadata." In applications in which 
the storage system is directly attached to a host that Implements this function itself, 
metadata modules are not needed and are accordingly not included in configurations 
intended for such applications. Since these applications and other storage system 
applications (e.g., HTTP sender, web cache protocol server, and FTP server 
applications) require a subset of the functionality needed for standalone file systems, 
the illustrated embodiment is configured as a standalone file system, but the invention 
is equally effective in direct-attach applications. The following description applies 
equally to systems configured for direct attachment 

The switch module 216 provides the command and data paths used to 
interconnect the other three module types and contains both the control and data 
switches. In this embodiment of the invention, module 216 is capable of passing a 
block of data, for example, two kilobytes, between art)itrary pairs of modules at 
approximate fixed time increments, for example, approximately every four 
microseconds. Each host Interface, disk Interface and metadata module operates In 
full duplex mode, thereby nabling it to transmit and receive simultaneously at the 
aforementioned rate thereby supporting a systenvlevel data bandwidth of up to N 
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gigabytes/second, with N the total number of host interface, disl< Interface and 
metadata modules In the system. 

The previously listed advantages of this architecture take the following more 
concrete fomis when applied to the storage system. First, host Interface modules are 
allowed to send incoming data to any available disk interface module for storage 
regardless of where that data might have been previously stored. This ability, in turn, 
distributes read accesses across the full complement of disks avoiding the inevitable 
hotspots encountered in conventional storage systems in which disks are partitioned 
into physical volumes and data must be directed to a specified volume. 

Second, additional metadata modules, disk interface modules and physical 
disks can be added at any time. Clients need not be aware that additional resources 
have been made available since knowledge of where data is physically stored is not 
visible to them. This allows the logical space allocated to clients to far exceed the 
physical space that Is cun-ently available. Physical disk space does not need to be 
added until clients begin to use up a significant portion of the available physical space, 
which Is typically much less than the allocated logical space. 

Third, additional host interface modules can be added at anytime to increase 
the connectivity available to the cun-ent clients or to add new clients. Since all host 
interface modules have equal access to all resources, traditional data segmentation 
and replication is not needed to provide access to an expanded set of clients. For the 
same reason, clients can transfer data to multiple disks in a single transfer; clients are. 
in fact, unconcerned about where that data is physically stored. 

A more detailed diagram of a host interface module is shown In Figure 3. Each 
host Interface module 300 Is composed of four major components: a central 
processing unit (CPU) complex 324, a data complex 318, an input/output (I/O) 
complex 302 for communicating with the host and a switch interface 352. The CPU 
complex 324. in turn, consists of a microprocessor CPU 332 with its associated level- 
one (internal) and level-two (external) caches 330. memory and I/O bus control logic 
334. local random-access memory (RAM) 326. content-addressable memory (CAM) 
338. A peripheral bus 336 provides access to the CAM 338. the data complex 318. 
the sw'itch interface 352. and, through an I/O buffer 328. to the I/O complex 302. A 
PCI bus 339 provides access over the data transfer bus 350 to the data complex 318 
and to two full-duplex channel adapters 340. 342 which connect to two full-duplex 
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10/100 megabit Ethernet channels called the Interprocessor Comnnunicatlon channels 
(IPCs) 346, 348 used to communicate with other modules in the system. 

The data complex 318 comprises a large (typically two-gigabyte), partty- 
protected data memory 322 supported with a memory controller 320 that generates 
the control signals needed to access the memory 322 over a 128-bit data bus 323 and 
Interface logic and buffers providing links to the I/O complex 302, the switch interface 
352 and, over the data transfer bus 350, to the CPU complex 324. The memory 
controller 320 responds to read requests from the other sections of the host interface 
module and fonwatds the requested infonnation to them, Similariy, it accepts write 
requests from them and stores the associated data in the specified locations in 
memory 322. 

The I/O complex 302 is used to communicate with the clients via ports 307-31 3. 
There are two versions of the I/O complex 302, one version supports four one-gigabit, 
full-duplex Ethemet ports and the second version supports four one-gigabit, full duplex 
Fibre Channel ports. The second of these versions is typically used for systems 
directly attached to hosts; the first version is used for network-attached storage 
systems and is a prefenred embodiment. Multiple protocols, Including SCSI, TCP/IP, 
UDP/IP. Fibre Channel, FTP. HTTP, bootp, etc., are supported for communicating 
over these ports between clients and the host interface modules. These protocols are 
interpreted at the host Interfaces 306-312. Commands (e.g., read or write a file, 
lookup a file or directory, etc.) are buffered In the local I/O memory 304 for access by 
the CPU software via bus 314. Data received from the clients, via ports 307-313, is 
sent to the data memory 322 where it is buffered pending further action. Similariy, 
data passed from the storage system 300 to clients is buffered in the data memory 
322 while the I/O complex 302 generates the appropriate protocol signals needed to 
transfer that data to the client that requested it. 

The switch Interface 352 contains a buffer memory 354 and associated logic to 
accept, upon command from the CPU software over the peripheral bus 336, 
commands to transfer data, via bus 357. from the data complex 318 to external 
modules. It buffers those commands and submits requests to the switch (216, Figure 
2) for access to the destinations specified In those commands. When a request is 
granted, the switch output logic 356 commands the memory controller 322 to read the 
specified locations in memory 322 and transfer the data to It to be fonwarded to the 
Intended destination. Similariy, the switch Input logic 358 accepts data from the switch 
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at the fiill switch bandwidth and forvards it, along with the accompanying address, to 
the data complex 318 viia bus 364. Data is transfened from the output logic 356 to the 
switch and fpom the switch to the input logic 358 using, in each case, four serial, one- 
gigabit/second connections 360, 362 giving the host interface module the ability to 
transmit, and simultaneously to accept, data at a rate of 500 megabytes/second. 
Similariy, the request and grant paths to the switch are also both implemented with 
serial one-gigabit/second links. 

When a request is received from a client over one of the Ethemet or Fibre 
Channels 307-313, the I/O complex 302 generates the appropriate communication 
protocol responses and parses the received paclcet of infonnation, directing the 
request to buffer 304 to await processing by the CPU software and any associated 
data to buffer 304 for subsequent transfer to the data memory 322. The processing 
steps taicen by the software running on the host interface module CPU 332 are 
illustrated in the flowchart shown in Figures 4A-4C. 

In Figure 4A, the process starts in step 400 and proceeds to step 402. where, 
the host interface receives a request from the client. The request always contains a 
file or directory "handle" that has been assigned by internal file system processing to 
each data object. This file system processing is typically done in the metadata 
module. The handle identifies that object and is sent to the client to be used when the 
client is making future references to the object. Associated with each such handle is 
an "inode" which is a conventional data stoicture that contains the object "attributes" 
(i.e., the object size and type, an identification of the users entitled to access it, the 
time of its most recent irodification, etc.) of the file or directory. Each inode also 
contains either a conventional map, called the "frnap", or a handle, called the tmap 
handle", that can be used to locate the frnap. The frnap identifies the physical 
locations, called the global physical disk addresses (GPDAs), of the component parts 
of the object indexed by their offsets from the starting address of that object. In step 
404. upon reading the request from the request buffer 304, the CPU software extracts 
the object handle fronri the request 

Next. In step 406. the CPU software queries the local CAM memory 338. using 
the extracted object handle as a key. to detemiine if the desired Inode Infomiation is 
already stored in host interface local memory 326. If the inode Infomnatlon is present 
in the memory 326. the CAM memory access results in a "hit" and the CAM memory 
338 returns the address In local memory 326 where the Inode InfonDatlon can be 
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found. In step 408, this address is then used to fetch the inode infonnation from the 
local memory 326. 

If the Inode Information Is not present In the local memory as Indicated by a 
CAM memory "miss", then, In step 410, the software uses the IPC links 346 and 348 
to contact the appropriate metadata module (which is Identified by the object handle) 
for the needed infomiatlon which is returned to it also over the IPC links. Once the 
software has located the inode (or a critical subset of the contents of the inode). it 
verifies that requested action is pennltted (step 412). If the action Is not permitted, an 
enx>r Is returned in step 414 and the process ends in step 415. 

Alternatively, If the requested action is permitted, then. In step 416, the CPU 
software determines which response is required. If stored data is to be read, the 
process proceeds, via off-page connectors 419 and 421 to step 418 shown in Figure 
4C where the CPU software detemnines whether the fmap or the fmap handle required 
to honor that request Is In the Inode. If the fmap itself is too large to be contained In 
the inode, the process proceeds to step 420 where the software again consults Its 
CAM 338, this time using the fmap handle as the key. to determine If the fmap pointed 
to by that handle is stored locally. If it Is not, the process proceeds to step 422, where 
the software extracts the GPDAfor the fmap ftiom the frnap handle and sends a 
request for the fmap, or for the next level of fmap pointers, over the IPC links 346, 348 
to the disk interface module identified by the GPDA, which returns the fmap, or the 
page containing the next level of fmap pointers, through the switch module and switch 
interface 352 to the host interface module data memory 322. The software can then 
access this Information over the data transfer bus 350. In step 424, the software 
checks the infomiatlon in the local data memory 322 to determine whether it has 
obtained the fmap. If not the process retums to step 420 and continues this process 
until It finds the ftnap and the GPDA of the data Itself, during each iteration of the 
process checking its CAM 338 at step 422 to detennine if the desired infonnation is 
cached in the data memory 322. 

When ttie software locates the GPDA of the desired data either through the 
process set forth In steps 420-424 or If it was determined that the fmap was in the 
inode In step 41 8, In step 426. the software again checks the. CAM 338 to determine if 
the data resides in the host Interface data memory 322. If the data Is not In the data 
memory 322, in St p428.th software sends a read request forth data to the disk 
interfac module to retri ve that data Identifi d by the GPDA. Once the data is In the 
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host interfiace data memory 322, in step 430 the software sends a response over the 
peripheral bus 336 via the I/O buffer 328 to the I/O complex 302 indicating the location 
in the data memory 322 where the desired Information resides, thereby enabling the 
I/O complex 302 to complete the transaction by returning the requested information to 
the client. The least-recently-used (LRU) replacement algorithm is used to manage 
the local memory 322 and data memory caches. The process then ends in step 432. 

If in step 416, a write operation is requested, the process proceeds, via off- 
page connectors 417 and 433, to steps 434-440 in Figure 4B. Prior to any writes to 
disl(, the disk interface module preallocates or assigns "allocation units" (AUs) of 
physical disic space to each host interfiace module. Each allocation unit consists of a 
512-kilobyte segment spread evenly across a plurality of (for example, four) dislcs. 
The disk interface module sends to each host interface module over an IPC channel 
346, 348. the logical addresses of the allocation units that have been set aside for it. 
The host interface module then uses these logical addresses to specify the location of 
an object. Accordingly, the GPDA assigned by a host interface module to identify the 
location of a particular object specifies the 'system parity group number" (SPGN), 
"zone" and offset within the zone where that object can be found. During initialization, 
the system determines the storage topology and defines a mapping associating 
SPGNs and specific disk interface modules. This level of mapping provides additional 
virtualization of the storage space, enabling greater flexibility and independence from 
the specific characteristics of the physical disks. The zone defines a particular region 
within a given SPGN. The disk module reserves certain zones for data represented 
by that allocation unit. The disk interface module also reserves certain zones for data 
that Is knovim to be frequently accessed, for example, metadata. It then allocates 
these zones near the center of the disk and begins allocating the rest of the space on 
the disk from the center outward towards the edges of the disk. Consequently, most 
of the disk activity is concentrated near the center, resulting in less head movement 
and faster disk access. 

if. in step 416. it is determined that the host interface module received a write 
request from the client, the process proceeds, via off-page connectors 417 and 433, to 
step 434 where the host interface module fonvards the request to the metadata 
module or other file system. In parallel, the software assigns the associated data, 
which is buffered by the I/O complex 302. to the appropriate prealiocated allocation 
unit in the data memory 322 and sends a request to tiie switch through the switch 
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interface 352 to enqueue th data for transmission, in, typically, two-kilobyte packets, 
to the corresponding disk interface module. 

Next, In step 436, the software then sends th GPDA(s) of the new location for 
that data over an IPC channel 346, 348 to the appropriate metadata module and 
broadcasts that same Infonnation over IPC channels 346, 348 to all other host 
Interface modules in the system so that they can update their copies of the fmap in 
question. Alternatively, the software can broadcast invalidate messages to the other 
host interface modules causing them to invalidate, rather than update, the associated 
fhiap. Then in step 438, the host interfece module waits for acknowledgements from 
the disk interface module and metadata module. Acknowledgements are sent to the 
host interface module over the IPC channels 346, 348 from the disk Interface rnodule 
when the data has been received and from the metadata module when rt has updated 
its copy of the fmap. When both acknowledgements have been received, in step 440, 
the CPU software signals the I/O interface 302 to send an acknowledgement to the 
client indicating that the data has been accepted and is secure. By "secure" it is 
meant that the data has been stored in two independent modules (the host interface 
module and a disk interface module) and the associated metadata updates either 
have also been stored in two modules (the host interface module and a metadata 
module) or a log of those updates has been stored on a second module. The process 
then ends in step 442 

The preallocation of allocation units has several significant advantages over the 
cun-ent state of the art in disk storage. In particular, the host interface module is able 
to respond to wrfte requests without having to wait for disk space to be allocated for it, 
allowing it to implement the request immediately and to ackno\Medge the write much 
more rapidly. In effect, the preallocation gives the host interface module direct 
memory access to the disk. This ability to respond quickly is also enhanced by the 
fact that the data write does not need to wait for the metadata update to be completed. 

As discussed below each disk interface module also maintains cached copies 
of allocation units. When a cached copy of an allocation unit in a disk interface 
module has been filled and written to disk, the disk interface module releases the 
cached copy, preallocating a new allocation unit, both on disk and in its cache, and 
sending the host interface module a message to that ffect over the IPC 346, 348. 
The disk interface module can then reuse th cache memory locations previously 
occupied by the released allocation unit. At any given time, each host interface 
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module has several allocation units preallocated for it by each disk interface module. 
Host interface modules select which allocation unit to use for a given write t>ased 
solely on the recent activity of the associated disi< Interface module. This enables the 
workload to be distributed evenly across all disks, providing the system with the full 
disk bandwidth and avoiding the serious perfomiance limitations that are frequently 
encountered in standard storage systems when multiple hosts attempt to access the 
same disk at the same time. 

In accordance with one aspect of the invention, the data assigned to a given 
allocation unit may come fipom multiple hosts and multiple files or even file systems. 
The only relationship among tiie various data items comprising an allocation unit is 
temporal: tiiey all happened to be written In tiie same time Interval, in many cases, 
this can offer a significant performance advantage since files, or portions of files, tiiat 
are accessed In close time proximity tend to be re-accessed also in dose time 
proximity. Thus, when one such file is accessed, ttie otiiers will tend to be fetched 
from disk at tiie same time obviating the need for subsequent disk accesses. On the 
other hand, this same process obviously gives rise to potential finagmentation, witii the 
data associated witii a given file ending up stored in multiple locations on multiple 
disks. Procedures to mitigate the possible deleterious effect of fragmentation when 
those files are read are discussed below. This technique for allowing data to be 
stored anywhere, witiiout regard to its content or to tiie location of its prior incarnation, 
allows for superior, scalable performance. 

As in any storage system, it is necessary to identify disk sectors Oiat contain 
data that is no longer of Interest, elttier because the file in question has been deleted 
or because it has been written to another location. This is accomplished In tiie current 
Invention by maintaining a reference count for each page stored on disk. When a 
page is written to a new tocation. new fttiap entries must be created to point to tfie 
data as described in the preceding paragraphs. Until tiie pages containing tiie old 
(map entries have been deleted, other pages pointed to by other entries on tiiose 
same pages will now have an additional entry pointing to them. Accordingly, their 
reference counts must be Incremented. When an fmap page is no longer needed (i.e., 
when no higher-level fmap points to It) It can be deleted and tiie reference counts of 
tiie pages pointed to by.entiles in the deleted finap page must be decremented. Any 
page having a reference count of zero then becomes a "free" page and the 
corresponding disk locations can be reused. 

12 
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This procedura allows volumes to be copied virtually Instantaneously. Volume 
copies are commonly used to capture the state of a file system at a given Instant. To 
effect a volume copy, it Is only necessary to define a new volume with pointers to the 
fmaps of the files that are being copied. When a page in a copied file is to be 
modified, the new fmap entries point to the new location while the old fmap entries 
point to the original, static, version of the file. As a result, unmodified pages are now 
pointed to by more than one fmap entry and their reference counts are incremented 
accordingly to prevent their being deleted as long as any copy of the volume is still of 
interest. 

Figure 5 Illustrates In more detail the construction of a disk interface module 
500. All disk interface modules contain the same construction and are 
interchangeable. The construction of each disk interface module is similar to the 
construction of a host interface module shown in Figure 3 and similar parts have been 
given corresponding numeral designations. These parts operate in a fashion identical 
with their cpn-esponding counterparts in Figure 3. For example, data memory 322 
con-esponds to data memory 522. The major differences between the two modules lie 
in the I/O complex 502 and in the data complex 518. The I/O complex 302 in each 
host interface module is replaced in each disk interface module with a complex 502 
consisting of five one-gigabit, full-duplex Fibre Channel interfaces (504-515), each 
containing the logic needed to send data to and to retrieve data from disk drives from 
various manufactures over five Fibre channels 507-517. These Fibre Channel 
interfaces 504-515 are used to communicate with sets of five disks, each channel 
supporting up to 90 disks, enabling each disk Interface module 500 to control up to 
450 disks. Parity infonmation is stored along with the data, so twenty percent of the 
disk space is used for that purpose. However, each disk interface module can still 
mfinanp. nnariv 3^ terahvtes of data usina 73-ainabvte dis^k^. 
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hardware unit 521 that is dedicated to calculating the parity needed for protection of 
the integrity of data stored on disk. During a disk write operation, the memory 
controller 520 successively transfers each of a set of four blocks of data that is to be 
written to disk to the parity generator 521 which generates the exclusive-or of each bit 
in the first of these blocks with the con^sponding bit in the second block, the 
exclusive-or of these bits with the corresponding bits in the third block and the 
exclusive-or of these bits with their counterparts in the fourth block. This resulting 
exclusive-or block is then stored in memory 522 to be transferred, along with the data, 
to the five disks over five Independent channels 507-515. The size of the blocks is 
refenred to as the "stripe factor^ and can be set according to the application. The 
spedfic disk used to store tiie parity block Is a function of the allocation unit being 
stored. This allows the parity blocks to be spread evenly across tiie five disk channels 
507-515. 

The steps taken by the disk interface module CPU 532 in response to read and 
write requests are illustrated in the flowchart in Figure 6. The process begins in step 
600 and proceeds to step 602 in which a new event is received by the disk interface 
module. On detecting that a new event has occurred, i.e., that eitiier data has been 
received over the switch or a request has been received over the IPC, the CPU 
software in the target disk interface module determines the appropriate action in step 
604. If the event is a read request, the process proceeds to step 610 in which tiie 
CPU software checks the disk interface module CAM 538, using tiie GPDA provided 
in the request as a key, to determine if the desired object is cached in its local data 
memory 522. If the data is cached, tiie process proceeds to step 61 6, described 
below. 

Altemativeiy, if in step 610, it is determined that tiie data is not cached, the 
process proceeds to step 612 in which software sends a request to the I/O complex 
502 directing that the requested data be read, along with, typically, several adjacent 
disk sectors in anticipation of subsequent reads, and stored In an assigned location in 
tiie data memory 522. The number of addKional pages to be read is specified in the 
read request generated in the requesting host interface module, this number is 
detemiined from an examination of the type of file being read and other information 
gleaned by the host interface module from the attributes associated witii tiie file. The 
additional pages are cached in case they are subsequentiy needed and ovenA^ritten if 
ttiey are not. 

14 
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The CPU software then polls the I/O complex 502 to determine when the read 
Is complete as Illustrated in step 614. When the data Is located In the data memory 
522. either through a cache hit or by being transfen^d In from disk, In step 616, the 
software sends a message to the switch interface 552 thereby enqueuing the data for 
transmission to the requesting host interface module. While any data item cached in 
the disk interface module data memory 522 as the result of a write must also be 
cached in some host Interface module data memory, the data item is not necessarily 
cached in the memory of the host interface module making the read request. 
Similariy, data may be cached In a disk interface module due to a prior read from 
some host interface module other than the one making the cun-ent request 

If, In step 604, It is detemilned that a write request has been received, the 
process proceeds to step 606. Allocation units that have been preallocated to host 
interface modules are represented by resented cadie locattons in the disk interface 
module local memory 522 and by "free space" on disk, that Is, by sectors that no 
longer store any data of interest When the switch interface 552 receives write data 
from a host Internee module. It stores the data directly in the preassigned allocatfon 
unit space In Its data memory 522 and enqueues a message for the CPU software that 
the data has been received. Upon receiving the message, the software sends an 
acknowledgement over the IPC links 546, 548 to the appropriate host Interface 
module as shown in step 606 and enqueues the data for transfer to disk storage, via 
the I/O complex 502, as shown in step 608. The process then temnlnates in step 618. 

When disk bandwidth is available, or when space is needed to accommodate 
new data, the CPU software instmcts the I/O complex 502 to transfer to disk storage 
the contents of one allocation unit cached in the data memory 522. If possible, the 
CPU software selects an allocation unit that is already full to store to disk and, of full 
allocation units, it selects an allocation unit that is "relatively inactive." One typical 
method for performing this selection Is to select the allocation unit that has been least 
recently accessed (according to one of several well-known least-recently-used 
algorithms) but other criteria could also be used. For example, one of the pre- 
allocated allocation units may be selected at random. Altemativety, the switch unit 
could keep track of the length of queues of transactions awaiting access to the various 
disk modules. This information could then be communicated back the host Interface 
module and used to mak a dedslon as to which allocation unit to s lect based on 
actual disk activity. 

15 
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As previously noted, the parity-generation hardware 521 is used to form th 
parity blocks that are stored along with the data, therefore, in order to store one 
allocation unit, 128 kilobytes of data and parity infomnation are sent over each of the 
five channels 507-51 5 to five different disks. Once the data has been stored on disk, 
the software sends a message to the relevant host interface module over the IPC links 
546, 548 informing the host interface module that the contents of the allocation unit 
can now be released. However, those contents are not ovenA/ritten in either the host 
interface module or the disk interface module until the space is actually needed, 
thereby allowing for the possibirity that the infonmation might be requested again 
before it is expunged and hence can be retrieved without having to access physical 
disk storage. 

Note that since an allocation unit is written as a unit, the parity Infomiatlon 
stored along with each disk stripe never has to be read and updated, reducing by 
3/4ths the number of disk accesses that would othenvise be needed to store a single 
page. For example, in prior art systems, the prior contents of the page to be stored 
has to be read, the parity page has to be read and modified based on the change 
between the new and old contents of the page in question, and the new page and the 
parity page both have to be stored. In the inventive system, the only time a parity 
page nonnally has to be read is when the data on some sector fails the standard cyclic 
residue code (CRC) check always used to protect data stored on disk. In this event, 
the parity sector is read and, in combination with the three en-or-free sectors, is used 
to reconstruct the contents of the defective sector. 

As previously noted, the policy of writing data to arbitrary locations, while 
offering major performance advantages, can result in fragmentation of files that are 
only partially updated. Since each host interface module can use any allocation unit at 
its disposal, and, in fact, selects allocation units solely on the basis of tiie recent 
activity of the associated disks, files may well be split up among multiple disk Interface 
modules. This tendency toward fragmentation is mitigated by a write-back policy. 
That is, when a host Interface module reads a file that has been fragmented, it follows 
that read witin a write, placing all the file firagments, or all ttiat will fit, in ttie same 
allocation unit. The previously described technique for ensuring that newly written 
data and metadata are consistent is, of course, used with write-back operations as 
well. 
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Another potential inefficiency resulting from the "Nvrite anywhere" policy is that 
sections of allocation units are gradually replaced by more up-to-date versions written 
elsewhere, leaving holes In those allocation units that represent wasted disk space 
unless they are Identified and reused. Since the reference count technique described 
earlier allows those sections to be identified, they can, in fact, be reused. To make 
their reuse more efficient, the software running on the disk Interf'ace modules CPU 
532. as a background task, Identifies those allocation units having more than a 
predetermined percentage of unused space and sends the GPDAs of the still-valid 
sectors to a host interface module so that the vectors can be read and rewritten more 
compactly. 

The detailed construction of a metadata module 700 is shown in Figure 7. The 
metadata module 700 differs from the host interface module 300 and disk interface 
module 500 in two basic ways: The metadata module 700 has no i/O complex since it 
does not communicate with either clients or disks; and the data complexes present in 
the host interface modules and disk interface modules are eliminated and their large 
data memories replaced by relatively a small memory 754 that serves as store-and- 
fonvard buffer. Data destined to be stored through the switch output 756 and 
connections 760 Is first transferred, using a DMA engine 753, from the CPU's local 
memory 726 into the buffer memory 754 before being enqueued for transfer. 
Similariy, data received over the switch via connections 762 and switch input 758 is 
transferred from the input buffer 754 directly into preassigned locations in local 
memory 726. 

Since the local memory 726 in the metadata module 700 stores all data 
received over the switch, it is considerBbly larger than its counterpart in the host 
interface 300 and disk interface modules 500, normally comparable in size to the latter 
modules' data memories, 322 and 522, respectively. The local memory 726 is used 
primarily for caching inodes and fmaps. The other elements shown in Figure 7 are 
similar in function and implementation to the corresponding elements shown in 
Figures 3 and 5. 

The purpose of the metadata module 700 is to maintain the file system 
structure, to keep all inodes consistent and to foPA^ard cunrent inodes to host interface 
modules that request them. When a new file or directory is created, it is the 
responsibility of the metadata module to generate the assodated inode and to insert a 
pointer to it into a B-tree data structure used to map between inodes and GPDAs. 

17 
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Similarly, when a file or directory is deleted, the metadata modul muist delete its 
associated inode. as well as those of all its descendents. and nKxlify the B-tree data 
structure accordingly. 

When a host interface module receives a request from a client that requires 
inode information that the host interface module cannot find in its own local memory, it 
uses the IPC linlcs to query the metadata module associated with the file system in 
question. The steps taken by the software mnning on a metadata module CPU 732 to 
service a typical request are depicted in Figures 8A and 8B. 

In Figure 8A, the process begins in step 800 and proceeds to step 802 where a 
request is received by the metadata module. All requests to a metadata module for an 
object are accompanied by a handle that Includes an "Inode number* uniquely 
identifying the object, or the parent of the object, being requested. These unique 
Inode numbers are assigned by the file system to each of its files and directories. The 
handle used by a client to access a given file or directory includes the inode number, 
which is needed to locate the object's associated inode. In step 804. the metadata 
module checi^ its CAM 738 using that inode number as a key. if the Inode 
Infonnation is In the CAM 738, the process proceeds, via off-page connectors 815 and 
819, to step 816, discussed below. 

If the inode infonnation is not in the local memory 726, as indicated by a cache 
"miss," the CPU software then searches through an inode B-tree data structure in 
memory 726 to find the GPDA of the inode data as indicated in step 806. if the 
necessary B-tree pages are not present in local memory, the process proceeds to step 
808 where the software sends a message over IPC linlcs 746, 748 to the appropriate 
disk interface module requesting that a missing page be returned to it over the switch. 
The metadata module 700 then waits for a response finom the disl^ interface module 
(step 810.) 

In step 812. the CPU software examines either the cached data from step 806 
or the data retumed finom the request to the 6isk interface module in step 810 to 
detemnine if the data represents a leaf page. If not. the process returns to step 808 to 
retrieve additional inode Infomfiation. If the data does represent a leaf node, then the 
process proceeds, via off-page connectors 813 and 817, to step 814. Once the 
metadata module 700 has located the GPDA of the inode itself (the desired leaf node), 
in step 814, the metadata module 700 sends a request over the IPC links 746. 748 for 
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the page containing that inode. At this point, the fnode infonnation has been obtained 
from the CAM 738 In step 804 or by retrieving the Information In step 814. 

The process then proceeds to step 816 where a determination Is made, 
concerning the request. If the request received from the host Interface module was to 
return the handle associated with a named object In a directory having a given Inode 
number, the retrieved Inode Is that of the directory and the process proceeds to step 
820. 

To fulfill the request, the metadata module 700 must read the directory itself as 
shown in step 820. The CPU software first queries its CAM 738, using the directory's 
GPDA as a key, to determine If the directory Infomnatlon Is cached In Ks local memory 
726. If the desired information is present, then the process proceeds to step 818. If 
the directory, or the relevant portion of the directory is not cached, the software must 
again send a message over the IPC to the disk ihterface module storing the directory 
requesting that the directory Infonnation be retumed to it through the switch as 
indicated in step 822. Once it has access to a directory page, It searches the page to 
find the desired object. If the object is not found in step 824, the process returns to 
step 820 to obtain a new page. Eventually it locates the named object and its 
associated inode number. 

Finally, once the metadata module has located either the inode of the object 
specified by the handle or the inode of the named object, depending on the specific 
request, it forwards the requested infonnation on to the requesting host interface 
module as set forth In step 818. The process then ends in step 826, 

A detailed diagram of the switch module is shown in Figure 9. The switch 
module 900 is composed of three major components: a crossbar switch complex 906 
providing non-blocking, full-duplex data paths between arbitrary pairs of host interface 
modules, disk interface modules and metadata modules; an IPC complex 904 
composed of switches 942 for two sets of full-duplex, serial. 10/100 Ethernet channels 
938 and 940 that provide messaging paths between art}itrary pairs of modules; and a 
configuration management complex 902 Including system reset logic 924 and the 
system dock 908. The switch module is implemented as a redundant pair for 
reliability and availability purposes, however, only one of the pair Is shown In Figure 9 
for clarity. 

The I/O processor 954 in the crossbar switch complex 906 accepts requests 
from th switch int rfaces 356. 556 and 756 on the host interface modules, disk 
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interface modules and metadata modules, respectively over the request links and 
grants access over the grant links. Each module can have one request outstanding 
for every other module in the system or for any subset of those modules. During each 
switch cycle, the arbiter 950 pairs requesting modules with destination modules. The 
arbiter assigns weights to each requester and to each destination. These weights can 
be based on any of several criteria, e.g., the number of requests a requester or 
destination has in its queue, the priority associated with a submitted request, etc. The 
arbiter then sequentially assigns the highest weight unpaired destination to the 
unpaired requester having the highest weight among those requesting It. It continues 
this operation as long as any unpaired requester is requesting any, as yet, unpaired 
destination. 

The I/O processor 954 then sends each requesting module, over ttie 
appropriate grant linic. the identity of the module with which it has been paired and to 
which It can send a data packet during the next switch cyde. The art>iter 950 sets the 
crossbar switch 952 to the appropriate state to effect those connections. 

The switch 952 Itself consists of four sets of multiplexers, one multiplexer from 
each set for each destination, with each multiplexer having one input fiiom each 
source. Switch cycles are roughly four microseconds in duration, during which time 
two kilobytes of data are sent between each connected pair with a resulting net 
transfer rate of approximately 500 megabytes/second per connected pair. 

The function of ttie IPC switches 942 is to connect source and destination IPC 
ports 944 long enough to complete a given transfer. The standard IEEE 802.3 SNAP 
(sub-networi< access protocol) communication pre)tocol is used consisting of a 22-byte 
SNAP header followed by a 21 -byte message header, a data packet of up to 512 
bytes and a 32-bit cyclic residue code (CRC) to protect against transmission enxjrs. 

The configuration management complex 902 coordinates system boot and 
system reconfiguration following faults. To support the first of these activities, it 
implements two external communications links: one 936 giving access through the 
PCI bus 928 via full-duplex, serial. 10/100 Ethernet channel 932; and the otiier 912 
giving RS-232 access 914 through the peripheral bus 922. To support tiie second 
activity, it implements the reset logic 924 for the entire system. It also implements and 
distributes the system clock 908. 

The disclosed Invention has several significant fault-tolerant features. By virtue 
of ttie fact ttiat it is implemented witti multiple copies of identical module types and ttiat 



20 



wo 02/08899 



PCTAJSOl/20977. 



all of these modules have equal connectivity to all other modules, it can survive the 
failure of one or more of these modules by transfening the workload previously 
handled by any failed module to other modules of the same type. The switch fabric 
itself, of course, is a potential single point of failure since all inter-module 
communication must pass through it However, as mentioned in the previous section, 
the switch in the preferred implementation is implemented with two identical halves. 
During the initialization process, the two configuration management complexes 902 
communicate with each other, via the IPC channels, to determine if both are 
functioning property and to establish which will assume the active role and with the 
standby role. If both switch halves pass their self-diagnostic tests, both sets of IPC 
channels are used and the configuration management complexes 902 cooperate In 
controlling the system configuration and monitoring its health. Each switch half, 
however, supports the full data bandwidth between all pairs of modules, therefore only 
the active half of the switch Is used for this purpose. If one switch half becomes 
inoperative due to a subsequent failure, the configuration management complexes 
cooperate to Identify the faulty half and, if It is the half on which the active 
configuration manager resides, transfer that role to the former standby half. The 
surviving configuration manager communicates the conclusion to the other system 
modules. These modules then use only the functioning half of the switch for all further 
communication until notified by the configuration manager that both halves are again 
functional. Although the IPC bandwidth is halved when only one switch half is 
operational, the full data bandwidth and all other capabilities are retained even under 
these circumstances. 

Several complementary methods are used to identify faulty modules, induding 
(1 ) watchdog timers to monitor the elapsed time between the transfer of data to a 
module and the acknowledgement of that transfer and (2) parity bits used to protect 
data while it is being stored in memory or transferred from one point to another. Any 
timeout or parity violation tiiggers a diagnostic program In the affected module or 
modules. If the violation occun^ed in the transfer of data between modules, the fault 
could be in the transmitting module, the receiving module or in the switch module 
connecting the two so the diagnostic routine Involves all three modules checking both 
themselves and their ability to communicate witii each other. Even if tiie diagnostic 
program does not detect a permanent fault, the event is logged as a transient. If 
transient vent recurs with a frequency exceeding a settable parameter, ttie module 
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Involved In the greatest number of such events is taken off line and the failure treated 
as pemianent, thereby triggering manual intervention and repair. If transients 
continue, other modules will also be taken off line as a consequence until the fault Is 
isolated. 

Byte parity is typically used on data stored in memory and various well-known 
forms of vertical parity checks and cyclic-residue codes are used to protect data 
during transfer. In addition, in the storage system embodiment described here, data 
tags consisting of 32-bit vertical parity check information on each data page are stored 
on disk separately from the data being protected. When data is retrieved from disk, 
the tag Is also retrieved and appended to the data. The tag is then checked at the 
destination and any discrepancy flagged. This provides protection not only from 
transmission errors but also from disk enors that result In reading the wrong data (or 
the wrong tag). This latter dass of emors can result, for example, from an addressing 
enor in which the wrong sector is read from disk or from a write current failure in which 
old data is not ovenvritten. 

Another important fault-tolerant feature of the storage system embodiment of 
the invention is the requirement that all data and metadata be stored on at least two 
different modules or on parity-protected disk before the receipt of any data is 
acknowledged. This guarantees that the acknowledged data will still be available 
following the failure of any single module. Similariy, data stored on disk is protected 
against any single disk failure, and against any single disk channel failure, by 
guaranteeing that each data block protected by a parity block is stored on a different 
physical disk, and over a different disk channel, from all other blocks protected by the 
same parity block and from the disk storing the parity block Itself. 

Finally, the fact that all disks are dual-ported to two different disk interface 
modules guarantees that data can still be retrieved should any one of those disk 
Interface modules fall. Following such an event and the resulting reconfiguration, all 
subsequent accesses to data stored on the affected disks are routed through the 
surviving disk interface module. While this may result in congestion because the 
surviving disk interface module is now servicing twice as many disks, It retains full 
accessibility. In addition, the previously described load-balancing capability of the 
system will Immediately begin redistributing the woridoad to alleviate that congestion. 

Similar protection against host interface module failures can be achieved by 
connecting clients to more than one host Interface module. Since all host interface 
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modules have full access to all system resources, any client can access any resource 
through any host interface module. Full connectivity Is retained as long as a client is 
connected to at least one functioning host interface module and, connection to more 
than one host interface module provides not only protection against faults, but also 
increased bandwidth into the system. 

The architecture described in the previous paragraphs exhibits several 
significant advantages over current state-of-the-art storage system architectures: 

1 ) It is highly scaleable. Host interface modules, disk interface modules and 
metadata modules can all be added independently as needed and their 
numbers can be independently increased as storage throughput or capacity 
demands increase. A system using a 16-port crossbar switch, for instance, can 
support any combination of host interface modules, disic interface modules and 
metadata modules up to a total of 16. This would allow a system to be 
configured, for example, to give 32 directly connected clients access to over 40 
terabytes of data (using 36-gigabyte disks) supported by two metadata 
modules. Obviously, even larger configurations can be realized with larger IPC 
and wider crossbar switches. 

2) Since writes can be directed to arbitrary disk interface modules, demand can be 
equalized across all disk resources, ensuring that throughput will increase 
neariy llneariy with the number of disk interface modules In the system. 
Further, writes can take place in parallel with fmap updates thereby decreasing 
the latency between the initiation of a data write and the acknowledgement that 
it has been accepted. Since both the data and the metadata assodated with a 
new write are always stored in two independent places before that write is 
acknowledged, write acknowledgements can be Issued before data Is actually 
stored on disk while still guaranteeing that the data is secure. 

3) Relegating metadata operations to modules designed for that purpose not only 
enables faster metadata processing but, in addition, allows the host interface 
and disk interface modules to be structured as efficient data pipes, with the bulk 
of local memory partitioned as a bi-directional buffer. Since the client's 
communication protocol is temiinated in the host interface module I/O complex, 
the bulk of data passing through this data memory does not need to be 
examined by th host interface module CPU software. 
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Alttibugh an exemplary embodiment of the invention lias been disclosed, it will 
be apparent to those skilled In the art that various changes and modifications can be 
made which will achieve some of the advantages of the invention without departing 
from the spirit and scope of the Invention. For example, it will be obvious to those 
reasonably skilled In the art that, although the description was directed to particular 
embodiments of host interface modules, disk Interface modules, metadata modules 
and switch modules, that other designs could be used in the same manner as that 
described. Other aspects, such as the specific circuitry utilized to achieve a particular 
function, as well as other modifications to the inventive concept are intended to be 
covered by the appended claims 

What is claimed is: 
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Claims 



1 1 . Apparatus for providing high-performance, scaleable data processing and 

2 storage services to a client from a plurality of resources, comprising 

3 an access interface module which receives requests for service from the 

4 client and selects a subset of the plurality of resources to provide the requested 

5 service and distribute the workload across the plurality of resources; and 

6 a switch fabric for temporarily connecting the access interface module to 

7 the selected subset of the plurality of resources for providing the service to the 

8 client. 

1 2. The apparatus of claim 1 wherein the access interface module selects the 

2 subset of the plurality of resources based on the relative demand placed on the 

3 subset of resources. 

1 3. The apparatus of claim 1 wherein the switch fabric comprises a control switch 

2 fabric for transfem'ng control information and a separate data switch fabric for 

3 transferring data, 

1 4. The apparatus of claim 3 wherein the control switch fabric is optimized for 

2 transfem'ng control infomiation and the data switch fabric is optimized for 

3 transferring data. 

1 5. The apparatus of daim 3 wherein the request for service includes control 

2 information and data and wherein the access interface module separates the 

3 control information and the data and transfers the data to the selected subset of 

4 resources over the data switch fabric. 

1 6. The apparatus of claim 3 wherein the data switch fabric comprises a non- 

2 blocking crossbar switch for data transfer and the control switch fabric 

3 coniprises an Ethernet switch for control Information transfer. 

1 7. The apparatus of dalm 1 further comprising a resource module connected to 

2 the plurality of resources for generating preallocation infomiation that 
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3 preallocates services from th plurality of resources in order to evenly distribute 

4 a workload across the plurality of resources. 

1 8. The apparatus of claim 7 wherein the switch fabric connects the access 

2 interface module to the resource module so that the resource module can 

3 transfer the preallocation information to the access internee module. 

1 9. The apparatus of claim 8 wherein the access interface module selects a subset 

2 of the plurality of resources based on the preallocation information. 

1 10. The apparatus of claim 1 wherein the access interface module comprises a 

2 data memory which temporarily stores infomiatlon transferred between the 

3 access interface module and the selected subset of the plurality of resources. 

1 11. The apparatus of claim 1 further comprising a plurality of access Interface 

2 modules each access interface module receiving service requests from a 

3 plurality of clients. 

1 12. A disk-based storage system for providing high-perfomnance, scaleable storage 

2 services to a client from a plurality of disks, comprising 

3 a disk interface module connected to the plurality of disks for controlling 

4 data stored on the plurality of disks; 

5 a host interface module which receives requests for storage service from 

6 the client and selects a subset of the plurality of disks to provide the requested 

7 storage and distribute the woridoad across the plurality of disks; and 

8 a switch fabric for temporarily connecting the host interface module to 

9 the resource module for providing the storage service to the client. 

1 1 3. The storage system of claim 1 2 wherein the switch fabric comprises a control 

2 switch fabric optimized for transferring control information and a separate data 

3 switch fabric optimized for transferring data. 

1 14. The storage system of claim 13 wherein the request for service includes control 

2 infomnation and data and wherein the host interfac module separates the 
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3 control information and the data and transfers the data to the selected subset of 

4 resources over the data switch fabric. 

1 15. The storage system of claim 1 3 wherein the data switch fabric comprises a 

2 non-blocking crossbar switch for data transfer and the control switch fabric 

3 comprises an Ethernet switch for control Infomriation transfer. 

1 1 6. The storage system of claim 1 2 wherein the disk interface module generates 

2 preallocation jnfom^ation that preallocates physical storage in the plurality of 

3 disks in order to evenly distribute data across the plurality of disks. 

1 1 7. The storage system of claim 1 6 wherein the physical storage in the plurality of 

2 disks Is divided into zones and the disk interface module preallocates selected 

3 zones to frequently-accessed data, wherein the selected zones are selected in 

4 order to decrease disk access time. 

1 18. The storage system of claim 1 6 wherein the host interface module logically 

2 maps data items to be stored into allocation units preallocated to the host 

3 interface module by the disk interface modules. 

1 19. The storage system of claim 1 2 wherein the host interface module comprises a 

2 first data memory and the resource module comprises a second data memory 

3 and wherein the first and second data memories temporarily store infonmation 

4 transfenred between the host interface module and the disk Interface module. 

1 20. The storage system of claim 1 2 further comprising a plurality of host interface 

2 modules, each host interface module receiving service requests from a plurality 

3 of clients. 

1 21 . The storage system of claim 12 further comprising a plurality of disk interface 

2 modules connected to the plurality of disks. 

1 22. The storage system of daim 21 wherein the host interface module directs 

2 writes to disk interface modules of data based on the relativ demand placed 
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3 on those modules without regard to either th contents of associated disks or 

4 the location of other copies of the data. 

1 23. The storage system of claim 22 further comprising a metadata module that is 

2 connected to the switch fabric and assigns each data object an identifying 

3 handle. 

1 24. The storage system of claim 23 wherein the host interface module and the 

2 metadata module store metadata information associated with each data object 

3 in the plurality of disks separately from, and independently of, data associated 

4 with each data object. 

1 25. The storage system of claim 24 wherein th6 handle contains the location of the 

2 metadata in the plurality of disks. 

1 26. The storage system of claim 23 wherein the metadata module is dedicated 

2 exclusively to fetching, caching and manipulating file and directory attributes of 

3 each data object thereby allowing data paths to be optimized to maximize data 

4 flow. 

1 27. The storage system of claim 1 2 wherein the disk interface module preallocates 

2 storage in the plurality of disks in allocation units. 

1 28. The storage system of claim 27 wherein the host interface module selects a 

2 subset of the plurality of disks from the preallocated storage. 

1 29. The storage system of claim 27 wherein the host interface module reads a file 

2 from the plurality of disks and writes the file to disk in a single allocation unit. 

1 30. The storage system of claim 27 wherein the disk interface module comprises a 

2 data memory for temporarily storing a data file that is to be written to disk and a 

3 parity generator for generating parity information on the data file stored in the 

4 datamemory prior to the transfer of the data file to th disk. 
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1 31 . The storage system of daim 30 wherein data and Its associated parity are 

2 stored on disk in their entirety In order to avoid reading previously stored data 

3 to deteonine data modifications and re-calculating parity. 

1 32. The storage system of claim 1 2 wherein the host interface module stores data 

2 to disl< without regard to where any earlier versions of that data were previously 

3 stored and without regard to the file and file system to which the data belongs. 

1 33. The storage system of claim 1 2 wherein the data to be stored on the plurality of 

2 disks IS arranged in a plurality of data pages and wherein the storage system 

3 comprises a reference counter for maintaining reference counts on each data 

4 page so that unused disk space can be readily identified. 

1 34. The storage system of claim 12 in which the modules are interconnected using 

2 a high-speed, non-blocking, crossbar switch for transferring data. 

1 35. The storage system of claim 12 In which separate, serial Interprocessor 

2 Communication (IPC) channels are used to transfer metadata between pairs of 

3 modules, thereby enabling the crossbar switch to be used at maximum 

4 efficiency for transfening data and allowing memory elements in the extemal 

5 interface to be partitioned into dedicated incoming and outgoing data buffers. 

1 36. A fault-tolerant computer system for providing scaleable data processing and 

2 storage services to a client ftom a plurality of storage resources, comprising 

3 a plurality of identical resource interface modules connected to the 

4 storage resources; 

5 a plurality of identical access interface modules which receive requests 

6 for service from the client and select a subset of the plurality of resource 

7 interface modules to provide the requested service and distribute the workload 
. 8 across the plurality of storage resources; and 

9 a switch for temporarily connecting one of the plurality of access 

10 interface modules to the selected subset of the plurality of resource interface 

1 1 modules for providing the sen^ice to the client, the switch being constructed in 
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12 two Identical halves which are Interconnect d so that a failure In one switch half 

13 does not make the computer system inoperative. 

1 37. The computer system of claim 36 wherein one switch half is designated as 

2 active and the other switch half is designated as standby and the active switch 

3 half Is used to temporarily connect one of the plurality of access interface 

4 modules to the selected subset of the plurality of resource Interface modules. 

1 38. The computer system of claim 37 wherein the active switch half and the 

2 standby switch half exchange roles if the active switch half falls. 

1 39 . The computer system of claim 36 wherein each storage resource Is connected 

2 to at least two resource interface modules so that a failure in any resource 

3 Interface module does not prevent access to the each storage resource. 

1 40, The computer system of claim 36 wherein each access interface module can 

2 assume the workload of any of the plurality of access interface modules so that 

3 a failure In any access interface module can be bypassed by assigning the 

4 workload of the failed module to another of the access interface modules. 

1 41 . The computer system of claim 36 wherein each resource interface module can 

2 assume the workload of any of the plurality of resource interface modules so 

3 that a failure in any resource interface module can be bypassed by assigning 

4 the woricload of the failed module to another of the resource interface modules. 

1 42. The computer system of claim 36 wherein data is stored on one of the plurality 

2 of storage resources and the computer system further comprises a parity 

3 generator which computes a data tag, Including parity infonnation, from the 

4 data and stores the data tag on the plurality of storage resource apart from the 

5 data. 

1 43. The computer system of claim 36 wherein data is stored on one of the plurality 

2 of storage resources and an acknowledgement is returned to the client after the 
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3 data has been stored and wherein the data Is stored on at least two separate 

4 storage resources before the acknowledgement is returned to the client. 

1 44. A method for providing high-performance, scaleable data processing and 

2 storage sen/ices to a client from a plurality of resources, the method comprising 

3 (a) pfx>Vidlng an access Interface module which receives requests for 

4 service from the client; 

6 (b) using the access Interface module to select a subset of the plurality of 

6 resources to provide the requested service and distribute the workload 

7 across the plurality of resources; and 

8 (c) using a switch fabric to temporarily connect the access Interface module 

9 to the selected subset of the plurality of resources for providing the 

1 0 service to the client 

1 45. The method of claim 44 wherein step (b) comprises selecting the subset of the 

2 plurality of resources based on the relative demand placed on the subset of 

3 resources. 

1 46. The method of claim 44 wherein step (c) comprises: 

2 (c1 ) using a control switch fabric for transferring control infomiation; and 

3 (c2) using a separate data switch fabric for transfening data. 

1 47. The method of daim 46 wherein step (c1 ) comprises optimizing the control 

2 switch fabric for transfening control infomiation and step (c2) comprises 

3 optimizing the dafa switch fabric for transfem'ng data. 

1 48. The method of claim 46 wherein the request for service includes control 

2 Information and data and wherein step (b) comprises separating the control 

3 information and the data and step (c) comprises transfem'ng the data to the 

4 selected subset of resources over the dafa switch fabric. 

1 49. The method of claim 46 wherein step (c1 ) comprises using a non-blocking 

2 crossbar switch for data transfer and step (c2) comprises using an Ethernet 

3 switch for control Infonnation transfer. 
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The method of claim 44 further comprising: 

(d) providing a resource moduie connected to the plurality of resources; and 

(e) using the resource module to generate preallocation infonnatlon that 
preallocates services from the plurality of resources in order to evenly 
distribute a workload across the plurality of resources. 



1 51 . The method of claim 50 wherein step (c) comprises connecting the access 

2 interface module to the resource module so that the resource module can 

3 transfer the preallocation infoimation to the access interface module. 

1 52. The method of claim 51 wherein step (b) comprises selecting a subset of the 

2 plurality of resources based on the preallocation infomnation. 

1 63. The method of claim 44 wherein step (b) comprises temporarily storing 

2 infomriation transfen-ed between the access interface module and the selected 

3 subset of the plurality of resources. 

1 54. The method of claim 44 wherein step (a) further comprises providing a plurality 

2 of access interface modules each access interface module receiving service 

3 requests from a plurality of clients. 

1 55. A method for providing high-perfonnance, scaleable storage services to a client 

2 from a plurality of disks, comprising 

3 (a) providing a disk Interface module connected to the plurality of disks for 

4 controlling data stored on the plurality of disks; 

5 (b) providing a host interface module which receives requests for storage 

6 service from the client and selects a subset of the plurality of disks to 

7 provide the requested storage and distribute the workload across the 

8 plurality of disks; and 

9 (c) using a switch fabric to temporarily connect the host interface module to 

10 the resource module for providing the storage service to the client. 



1 56. The method of claim 55 wherein step (c) comprises: 
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2 (c1 ) using a control switch fabric optimized for transferring control 

3 information; and 

4 (c2) using a separate data switch fabric optimized for transfemng data. 

1 57. The method of claim 56 wherein the request for service includes control 

2 infomiation and data and wherein step (b) comprises separating the control 

3 information and the data and step (c) comprises transfening the data to the 

4 selected subset of resources over the data switch fabric. 

1 58. The method of claim 56 wherein step (c1 ) comprises using a non-blocking 

2 crossbar switch for data transfer and step (c2) comprises using an Ethernet 

3 switch for control infomiation transfer. 

1 59. The method of claim 55 wherein step (a) comprises generating preallocation 

2 infomiation that preallocates physical storage in the plurality of disks in order to 

3 evenly distribute data across the plurality of disks. 

1 60. The method of claim 59 wherein the physical storage in the plurality of disks is 

2 divided into zones and step (a) further comprises preallocating selected zones 

3 to frequently-accessed data, wherein the selected zones are selected in order 

4 to decrease disk access time. 

1 61 . The method of claim 59 wherein step (b) comprises logically mapping data 

2 items to be stored into allocation units preallocated to the host interface module 

3 by the disk Interface modules. 

1 62. The method of claim 59 wherein the host interface module comprises a first 

2 data memory and the disk interface module comprises a second data memory 

3 and wherein step (a) comprises using the first data memory to temporarily store 

4 information transferred from the host interface module to the disk interface 

5 module and step (b) comprises using the second data memory to temporarily 

6 store information received by the disk interface module from the host interface 

7 module. 
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1 63. The method of daim 55 wherein step (b) further comprises providing a plurality 

2 of host interface modules, each host interface module receiving service 

3 requests from a plurality of clients. 

1 64. The method of claim 55 wherein step (a) comprises providing a plurality of disk 

2 interface modules connected to the plurality of disks. 

1 65, The method of claim 64 wherein the host interface module directs writes to disk 

2 interface modules of data based on the relative demand placed on those 

3 modules without regard to either the contents of associated disks or the 

4 location of other copies of the data. 

1 66. The metfiod of claim 65 further comprising 

2 (d) providing a metadata module that is connected to the switch fabric and 

3 assigns each data object an identifying handle. 

1 67. The method of claim 66 further comprising: 

2 (f) using the host interface module and tiie metadata module to store 

3 metadata infomiation associated with each data object in the plurality of 

4 disks separately from, and independentiy of, data associated with each 

5 data object. 

1 68. The metiiod of claim 66 wherein the handle contains the location of the 

2 metadata in the plurality of disks. 

1 69. The metiiod of claim 66 wherein tiie metadata module is dedicated exclusively 

2 to fetching, caching and manipulating file and directory attributes of each data 

3 object thereby allowing data patiis to be optimized to maximize data flow. 

1 70. The metiiod of claim 55 wherein step (a) comprises using the disk interface 

2 module to preallocate storage in the plurality of disks in allocation units. 

1 71 . The method of claim 70 wherein step (a) comprises using th host interface 

2 module to s lect a subset of ttie plurality of disks from the preallocated storage. 
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1 72. The method of claim 70 wherein step (a) comprises using the host interface 

2 module to read a file from the plurality of disks and write the file to disl< in a 

3 single allocation unit. 

1 73. The method of claim 70 wherein step (a) comprises using a data memory for 

2 temporarily storing a data file that is to be written to disk and using a parity 

3 generator to generate parity information on the data file stored in the data 

4 memory prior to the transfer of the data file to the disk. 

1 74. The method of claim 73 wherein data and its associated parity are stored on 

2 disk In their entirety in order to avoid reading previously stored data to 

3 detenr^ine data modifications and re*calculating parity. 

1 75. The method of claim 55 wherein step (b) comprises using the host interface 

2 module to store data to disk v\rtthout regard to where any eariier versions of that 

3 data were previously stored and without regard to the file and file system to 

4 which the data belongs. 

1 76. The method of claim 55 wherein the data to be stored on the plurality of disks is 

2 anBnged in a plurality of data pages and wherein the method further comprises: 

3 (g) maintaining reference counts on each data page so that unused disk 

4 space can be readily identified. 

1 77, The method of claim 55 wherein step (c) comprises Interconnecting the 

2 modules using a high-speed, non-blocking, crossbar switch for transfening 

3 data. 

1 78. The method of claim 55 wherein step (c) further comprises using separate, 

2 serial Interprocessor Communication (IPC) channels to transfer metadata 

3 between pairs of module's, thereby enabling the crossbar switch to be used at 

4 maximum efficiency for transfening data and allowing memory elements in the 

5 external interface to be partitioned into dedicated incoming and outgoing data 

6 buffers. 
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1 79. A method for providing fault-tolerant scaleable data processing and storage 

2 services to a client from a plurality of storage resources, comprising 

3 (a) providing a plurality of identical resource interface modules connected to 

4 the storage resources; 

5 (b) providing a plurality of identical access interface modules which receive 

6 requests for service from the client and select a subset of the plurality of 

7 resource interface modules to provide the requested service and 

8 distribute the woridoad across the plurality of storage resources; and 

9 (c) using a switch for temporarily connecting one of the plurality of access 

10 Interface modules to the selected subset of the plurality of resource 

1 1 interface modules for providing the service to the client, the switch being 

12 constructed in two identical halves which are interconnected so that a 

13 failure in one switch half does not make the computer system 

14 inoperative. 

1 80. The method of claim 79 wherein step (c) comprises designating one switch half 

2 as active and designating the other switch half as standby and using the active 

3 switch half to temporarily connect one of the plurality of access interface 

4 modules to the selected subset of the plurality of resource interface modules. 

1 81 . The method of claim 79 further comprising: 

2 (d) exchanging the roles of the active switch half and the standby switch half 

3 if the active switch half fails. 

1 82. The method of claim 79 wherein step (a) comprises connecting each storage 

2 resource to at least two resource interface modules so that a failure In any 

3 resource Interface module does not prevent access to the each storage 

4 resource. 

1 83. The method of claim 79 wherein step (b) comprises providing access interface 

2 modules that can assume the workload of any other of the plurality of access 

3 interface modules so that a failure in any access interface module can be 
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4 bypassed by assigning the workload of the failed module to another of the 

5 access interface modules. 

1 84, The method of claim 79 wherein step (a) comprises providing resource 

2 interface modules that can assume the workload of any other of the plurality of 

3 resource Interface modules so that a failure in any resource interface module 

4 can be bypassed by assigning the workload of the failed module to another of 

5 the resource interface modules. 

1 85. The method of daim 79 wherein data Is stored on one of the plurality of storage 

2 resources and the method further comprises: 

3 (e) computing a data tag, including parity Infomiation, from the data; and 

4 (f) storing the data tag on the plurality of storage resource apart: from the 

5 data. 

1 86. The method of claim 79 wherein data is stored on one of the plurality of storage 

2 resources and an acknowledgement is retumed to the client after the data has 

3 been stored and wherein the method further comprises: 

4 (g) storing data on at least two separate storage resources before the 

5 acknowledgement Is retumed to the client. 
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